The Softmax Nonlinearity: Derivation Using Statistical Mechanics and Useful Properties as a Multiterminal Analog Circuit Element

We use mean-field theory methods from Statistical Mechanics to derive the "softmax" nonlinearity from the discontinuous winnertake-all (WTA) mapping. We give two simple ways of implementing "soft max" as a multiterminal network element. One of these has a number of important network-theoretic properties. It is a reciprocal, passive, incrementally passive, nonlinear, resistive multiterminal element with a content function having the form of informationtheoretic entropy. These properties should enable one to use this element in nonlinear RC networks with such other reciprocal elements as resistive fuses and constraint boxes to implement very high speed analog optimization algorithms using a minimum of hardware.